Skip to content

ci(agent-e2e): run on gpt-6-luna; skip live dall-e-3/tts-1 tests - #191

Merged
chrishagglund-ship-it merged 8 commits into
mainfrom
ci/agent-e2e-gpt-6-luna
Nov 17, 2026
Merged

chrishagglund-ship-it merged 8 commits into
mainfrom
ci/agent-e2e-gpt-6-luna

Conversation

@chrishagglund-ship-it

Copy link
Copy Markdown
Contributor

The OpenAI key is moving to the Orkes-Developers project, which only allows the gpt-5.6 / gpt-6 families. gpt-4o-mini also shuts down on 2027-01-14.

  • Set CONDUCTOR_AGENT_LLM_MODEL=openai/gpt-6-luna for every suite.
  • Suite 11 (LangGraph): build ChatOpenAI from that model, without temperature. Reasoning models reject temperature unless reasoning effort is "none".
  • Suite 7: skip the live DALL-E 3 and TTS-1 tests. dall-e-3 no longer exists at OpenAI, gpt-image-* is not enabled, and the project gets a 403 for tts-1.
  • Add an "LLM provider check" first step that calls the model directly, so a dead key or quota fails in seconds, not at the 45-minute timeout.
  • Add a "Show server errors" step that prints server.log errors to the job log on failure or cancel.

The OpenAI key is moving to the Orkes-Developers project, which only
allows the gpt-5.6 / gpt-6 families. gpt-4o-mini also shuts down on
2027-01-14.

- Set CONDUCTOR_AGENT_LLM_MODEL=openai/gpt-6-luna for every suite.
- Suite 11 (LangGraph): build ChatOpenAI from that model, without
  temperature. Reasoning models reject temperature unless reasoning
  effort is "none".
- Suite 7: skip the live DALL-E 3 and TTS-1 tests. dall-e-3 no longer
  exists at OpenAI, gpt-image-* is not enabled, and the project gets a
  403 for tts-1.
- Add an "LLM provider check" first step that calls the model directly,
  so a dead key or quota fails in seconds, not at the 45-minute timeout.
- Add a "Show server errors" step that prints server.log errors to the
  job log on failure or cancel.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Revert before merge. A server bisect: on 3.34.0-rc2,
test_hitl_approve_path stalled twice in the same place, and guardrail
matrix #9 timed out once. Run only those two tests, in a matrix against
the org-var server version and 3.33.0, the last server the full JS suite
passed on (2026-10-26).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Revert before merge. test_hitl_approve_path stays RUNNING after one
approval on both 3.34.0-rc2 and 3.33.0 with gpt-6-luna, but passed on
3.33.0 with gpt-4o-mini. After the wait, print every task in its
workflow, with full inputs and outputs for the LLM and HUMAN tasks, to
see what the model does after the approved publish_article call.

Run only that test, still on both servers. Guardrail matrix #9 passed
alone on both and goes back to normal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On gpt-6-luna, test_hitl_approve_path stalled on every run (3.33.0 and
3.34.0-rc2). After the approved publish_article call, the server sends
the instructions and the original user request again, after the tool
result. Read literally, that is a new request: luna wrote and published
a second article, which waits for a second approval the test never
gives. gpt-4o-mini treated the request as already done.

Keep the task in the user message only. Make the instructions say how
to do the work and that one successful publish_article ends the request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Revert before merge. Make the test's system prompt, tool description
and user message overridable (HITL_SYSTEM, HITL_TOOL_DESC, HITL_USER).
Run one matrix job per variant on the org-var server (3.34.0-rc2), in
place of the 3.33.0 vs rc2 legs. Both servers behaved the same.

- control:      current text
- user-stop:    stop rule in the user message, which the server
                re-sends after the tool results each turn
- tool-once:    tool description says it can be called at most once
                per chat session
- system-count: system prompt says to count publish_article calls and
                end the session when the count is above zero

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Revert before merge. The control prompt failed in 4 of 4 runs. Keep the
three passing variants (user-stop, tool-once, system-count) for repeat
runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Remove the temporary bisect, diagnostic and prompt-variant changes:
the workflow and suites go back to their state at 3379c12, and the full
suite runs again.

On gpt-6-luna, test_hitl_approve_path stalled on both 3.33.0 and
3.34.0-rc2. After the approved publish_article call, the server sends
the original user request again, after the tool results. Luna often
reads that as a new request and publishes a second article, which waits
for a second approval the test never gives.

Prompt variants tested in CI (passed / runs):
- "publish once; when it returns 'published' you are done": 0/4
- tool description "at most once per chat session":          2/4
- user message "if already published, do not publish again": 3/4
- system prompt: count earlier calls, stop if above zero:     4/4

Use the last one as the agent's instructions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The live image and TTS tests now run by default and are skipped only
when SKIP_IMAGE_GEN / SKIP_TTS is "true". agent-e2e.yml sets them,
because the Orkes-Developers OpenAI project has no image or TTS models.
Also keep "Show server errors" green when grep finds no match.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread e2e/test_suite16_streaming.test.ts Resolved

@ling-senpeng13 ling-senpeng13 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good — approving.

  1. Suite 16 HITL test (optional, but I'd recommend it): I think the real problem is the test itself, not the model. It approves only once, so it only passes if the model calls the tool exactly once. Even the same model won't always do that, since its output varies from run to run, so this test can turn flaky again later. That's also why the prompt had to get so specific, and I'd rather not rely on a prompt like that: it will likely break again with the next model. If the test approved every time it was asked, it wouldn't depend on how the model behaves, and the prompt could go back to normal.
  2. Nit: e2e/helpers.ts still defaults MODEL to gpt-4o-mini, which shuts down in January.

@chrishagglund-ship-it
chrishagglund-ship-it merged commit 3c7bbd5 into main Nov 17, 2026
51 of 56 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants